Machine and equipment assets are engineered to perform particular tasks as part of a process. For example, assets can include, among other things and without limitation, industrial manufacturing equipment on a production line, drilling equipment for use in mining operations, wind turbines that generate electricity on a wind farm, transportation vehicles, gas and oil refining equipment, and the like. As another example, assets may include devices that aid in diagnosing patients such as imaging devices (e.g., X-ray or MM systems), monitoring equipment, and the like. The design and implementation of these assets often takes into account both the physics of the task at hand, as well as the environment in which such assets are configured to operate.
Low-level software and hardware-based controllers have long been used to drive machine and equipment assets. However, the rise of inexpensive cloud computing, increasing sensor capabilities, and decreasing sensor costs, as well as the proliferation of mobile technologies, have created opportunities for creating novel industrial and healthcare based assets with improved sensing technology and which are capable of transmitting data that can then be distributed throughout a network. As a consequence, there are new opportunities to enhance the business value of some assets through the use of novel industrial-focused hardware and software.
Industrial manufacturing-based systems generate and store a significant amount of data on a continuous basis. For example, edge devices (equipment, machines, etc.) may repeatedly feedback data to a central cloud platform or other host environment over the course of a short period of time (e.g., seconds, minutes, hours, etc.) Before the raw data can be consumed by other systems and software, the data is processed, converted, and stored in an appropriate location for further consumption. Stream processing may be used to transfer large amounts of data from a data source such as the edge of an industrial environment to a target destination such as a host platform. Stream processing provides high-velocity and high-volume data transfer with very little latency.
A stream processing application typically includes a plurality of operators which each perform a processing event on an input stream and output the processed stream. A first operator may be used to read the source data while the last operator in the pipeline may write the processed data to the target destination. In addition, one or more intermediate operators can perform additional functions on the data as it is transferred from the data source to the target destination. However, if a processing speed of an operator lags behind the processing speed of other operators, a bottleneck may be created within the pipeline leading to inefficient stream processing. On the other hand, if data is entering the pipeline to fast, the entire stream processing application can shut down. However, situations which influence processing speed can be unexpected and difficult to predict. Therefore, a technology is needed which can adapt to dynamic changes in the flow of data between operators in a stream processing application.